ALMOST PRIME PYTHAGOREAN TRIPLES IN THIN ORBITS 



ALEX KONTOROVICH AND HEE OH 



Abstract. For the ternary quadratic form Q(x) = x 1 +y 2 —z 2 and a non-zero 
Pythagorean triple xo £ J? lying on the cone Q(x) = 0, we consider an orbit 
O = xoT of a finitely generated subgroup Y < SOq(Z) with critical exponent 
exceeding 1/2. 

We find infinitely many Pythagorean triples in O whose hypotenuse, area, 
and product of side lengths have few prime factors, where "few" is explicitly 
quantified. We also compute the asymptotic of the number of such Pythagorean 
triples of norm at most T, up to bounded constants. 
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1. Introduction 

1.1. The Affine Linear Sieve. In BGS06 , Bourgain, Gamburd, and Sarnak 
introduced the Affine Linear Sieve, which extends some classical sieve methods to 
thin orbits of non-abelian group actions. Its input is a pair (0,F), where 

(1) O is a discrete orbit, O — x • T, generated by a discrete subgroup T of a 
linear group G. It is called "thin" if the volume of T\G is infinite; and 

(2) F is a polynomial, taking integer values on O. 

Given the pair (0,F), the Affine Linear Sieve attempts to output a num- 
ber R = R(0,F) as small as possible so that there are infinitely many integers 
n G F (O), with n having at most R prime factors. 

A special case of their main result is the following. 
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• Hypotenuse is prime 
Hypotenuse is composite 



Figure 1. A thin orbit O of Pythagorean triples, sifted by hy- 
potenuse, F(x, y, z) = z. The darker points denote those triples 
whose hypotenuse is prime. 



Theorem 1.1 f[BGS06l IBGS08] ). Let G < GL„(M) be a Q-form o/SL 2 , and let 
r be a non-elementar^ subgroup of G GL„(Z). Let O be an orbit x r for some 
xo G Z™ \ {0} and F any polynomial which is integral on O. Then there exists a 
number 

R = R(0,F) < oo 

such that there are infinitely many x G O with F(x.) having at most R prime factors. 
Moreover the set of such x is Zariski dense in the Zariski closure of O . 

Remark 1.2. As described in jBGS081 §2], Lagarias gave evidence that the result 
above may be false if one drops the condition that T is non-elementary. 

For various special cases of (0,F), one can say more than just R < oo; one 
can give explicit, "reasonable" values of R(0,F). This was achieved with some 
restrictions in [Kon07, Kon09 , and it is our present goal to improve the results 
there in a more general setting. 

In order to remove local obstructions which would increase R for trivial reasons, 
we will impose the strong primitivity condition on (0, F). 

Definition 1.3. For a subset O C Z™ and a polynomial F{x\,--- ,x n ) taking 
integral values on 0, the pair (0, F) is called strongly primitive if for every integer 
q > 2 there is an x G O such that 



Remark 1.4. The weaker condition of primitivity requires the above for q prime. See 
BGS08, §2] for an example of (0,F ) which is primitive but not strongly primitive. 

To present a concrete number R(0, F), we will consider the quadratic form 



Recall that a discrete subgroup T < SL(2, IE) is elementary if and only if it has a cyclic 
subgroup of finite index. 



F(x)^0 (modg). 



Q(x) = x 2 +y 2 -z 2 . 
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Hence a non-zero vector x G Z 3 is a Pythagorean triple if Q(x) = 0. Let G — 
SOq(M) be the special orthogonal group preserving Q with real entries. For a 
discrete subgroup T of G, the critical exponent < <5r < 1 of T is defined to be the 
abscissa of convergence of the Poincare series: 

Lr(s) :=]T!l7ir 

for any norm || • || on the vector space M 3 (M) of 3 x 3 matrices. We remark that T 
is non-elementary if and only if Sr > 0. Moreover if T is finitely-generated, then T 
is of finite co- volume in G if and only if Sr — 1 [Pat 76], 

The detailed statement of our main result is given in Theorem 2.23 The following 
is a special case: 

Theorem 1.5. Let T < SOq(Z) be a finitely generated subgroup and set 

0:=(3,4,5)r. 

Let the polynomial F be one of 

the hypotenuse: F-^(x) := z; 

the "area": -PU( X ) : = T2 x Vi 

the product of coordinates : -Fc( x ) : — ^Q^yz. 
We assume that the pair (0,F) is strongly primitive and that 



5 > < 



'0.9992 


ifF = 




< 0.99995 


ifF = 




0.99677 

V 


ifF = 


F c - 



Then the following hold: 

(1) For infinitely many x € O, the integer F(x) has at most R = R(0,F) 
prime factors, where 

f 14 ifF = F n ; 
R=<25 ifF = F A ; 
I 29 ifF = F c . 



(2) We have 

#{x £ O : ||x|| < T, -F'(x) has at most R(0,F) prime factors} 



rpSr 



(logT)" 

where || • || is any norm on M 3 and the sieve dimension k is 

*fF = F n ; 
ifF = F A ; 
ifF = F c . 

In particular, the set of x £ O such that -F(x) has at most R(0,F) prime 
factors is Zariski dense in the cone Q = 0. 

Remark 1.6. The functions F A and Fc satisfy F (3, 4, 5) = 1; hence the pair (O, F) 
is strongly primitive regardless of the choice of the group T. For the hypotenuse, 
Fu, one must check, given T, that the pair (0,F) is strongly primitive. 



2 



Recall that / X g means that c" 1 ■ f(T) < g(T) < c ■ f(T) for some c > 1 and for all T > 1. 
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Remark 1.7. The above theorem was proved in [Kon09| assuming that T contains 
a non-trivial (parabolic) stabilizer of (3,4,5). In this case, the orbit O contains an 
injection of affine space, and hence standard sieve methods Iwa78 also produce 
integral points with few prime factors. Some of the most interesting cases which 
cannot be dealt with using standard methods and are now covered by our results 
are the so-called Schottky groups; these are groups generated by finitely many 
hyperbolic elements. 

1.2. A Counting theorem. In order to sieve almost primes in a given orbit, one 
must know how to count points on such orbits, which we obtain without assuming 
the arithmetic condition on L. 

Theorem 1.8. Let Q be any ternary indefinite quadratic form, G = SOq(R), and 
r < G a finitely generated discrete subgroup with Sr > 1/2. Let xo G M. 3 be a 
non-zero vector lying on the cone Q = such that the orbit O — Xo • T is discrete. 
Then there exist a constant cq > and some £ > such that as T — > oo, 

#{x G O : ||x|| < T} = cq ■ T s + 0(T 5 - C ). 

The norm II • II above is Euclidean. 



Remark 1.9. Let Nq denote the (unipotent) stabilizer of Xq in G. The Theorem 1.8 



was proved in [Kon09 under the further assumption that r n No is a lattice in No 

1.3. Expanding Closed Horocycles. The main difference between this paper 
and [Kon09 is the method used to establish counting theorems such as Theorem 



1.8 While |Kon09| uses abstract operator theory, in the present work we prove the 
effective equidistribution of expanding closed horocycles on a hyperbolic surface X, 
allowing not only X to have infinite volume, but also allowing the closed horocycle 
to be infinite in length. 

Let G — SL2 (M) and write the Iwasawa decomposition G = NAK with 

N=\n x =(\ fjrzGRl, A={a y =(^> °S):y>o\, (1.10) 



and K = S0 2 (M). 

We use the upper half plane EI = {z = x + iy : y > 0} as a model for the 

hyperbolic plane with the metric — — . The group G acts on H by fractional 

linear transformations which give arise all orientation preserving isometries of H: 



b\ az 



c d) Z cz + d 



for Im(z) > 0. We compute: 



n x a y {i) = x + iy. 

Let F < G be a finitely generated discrete subgroup with 5r > 1/2. Assume 
that the horocycle (r n N)\N is closed in X :— T\G, or equivalently the image of 
N(i) — {x + i : x G K} is closed in r\H under the canonical projection H — > L\H. 
Geometrically, this is isomorphic to either a line R or to a circle K/Z, depending 
on whether or not r n N is trivial. We push the closed horocycle (N n T)\N(i) 
in the orthogonal direction a y , and are concerned with its asymptotic distribution 
near the boundary, corresponding to y — > 0. 
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Figure 2. Equidistribution of expanding closed horocycles on a 
Schottky domain in the disk D and upper half plane EI models. 



Let X = r\H and consider the Laplace operator A = —y 2 (d xx + d yy ). By 
Patterson |Pat76j and Lax-Phillips |LP82j . the spectral resolution of A acting on 
L 2 (X) consists of only finitely many eigenvalues in the interval [0,1/4), with the 
smallest given by Ao = #r(l — #r)- Denote the point spectrum below 1/4 by 

< A < Ai < ••■ < A fc < 1/4. 

Let (j>o, . . . , ipk be the corresponding eigenfunctions, normalized by ||0j||2 = 1- Let 
sj > 1/2 satisfy Xj — sj(l — Sj), j = 0, 1, 2, . . . , k, so that s = S. 

Theorem 1.11. Fix notation as above and assume that (Tf)N)\N is closed. Then 
for any ifr £ C^°(F\H), 

J ifr{x + iy) dx = ^{ip,(fr j ) J (frj(x + iy) dx + O e (y^-i^-^- s ), 

as y — > 0. Here the implied constant depends only on a Sobolev norm of ifr, and on 
e > which is arbitrary. 

Moreover, the integrals above converge, and satisfy 

J (frj(x + iy) dx ~ c.j ■ y 1 ~ Sj , as y — > 0, 
n I e(;vnr)\A' 
where Co > 0, and C\, • • • , C/. G R. 

Remark 1.12. If T is a lattice, then the closedness of TnN\N implies that TnN\N 
is compact. In this case, Sarnak }Sar811 proved the above result allowing ifr £ 
C^°(r\G) (that is, not requiring K-tixed), and with a best possible error term of 

If 2 

in place of our weaker bound 

1.4. Bounds for Automorphic Eigenfunctions. The proof of Theorem |1.11| 
requires control over the integrals of the eigenfunctions <frj , which a priori are only 
square-integrable. For the base eigenfunction, one has extra structure coming from 
Patterson theory [Pat 76 j which makes this control possible. But for the other 
eigenfunctions, this analysis fails. Nevertheless, the problem of obtaining such 
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control was solved in the first-named author's thesis |Kon07j . The statement is the 
following (see the Appendix as well). 



Theorem 1.13 f[ Kon07] ). Fix notation as in Theorem \l . 1 1\ and assume that the 
closed horocycle (N f]T)\N is infinite. Let <pj € L 2 (r\H) be an eigenfunction of 
eigenvalue Xj = Sj(l — s,) < 1/4 with Sj > 1/2. Then 

4>j(n x a y ) <0 3 

as \x\ — > oo and y — > 0. 



y 2 



1.5. Organization of the Paper. In <|2]we give some background and elaborate 
further on Theorem|1.5| For the reader's convenience, in the Appendix we reproduce 



the proof of Theorem 1.13 from |KonQ7], since this reference is not readily available. 



Equipped with such control, the proof of Theorem 1 1 . 1 1 1 follows with minor changes 
from the one given for one dimension higher in )KO08l. We sketch the argument 
in f|3j and use it to prove Theorem |2.5| in ^4] In f|5j we verify the sieve axioms in 
Theorem 1 2 . 1 9| and conclude Theorem |2.23[ At the end of f|5] we derive the explicit 
values of i?, in particular proving Theorem |1.5| 



Acknowledgments. The authors wish to express their gratitude to Peter Sarnak 
for many helpful discussions. 

2. Background and More on Theorem 11.51 



In this section, we elaborate on Theorem |1.5| Let Q be a ternary rational 
quadratic form which is isotropic over Q. Let F < SOq(Z) be a finitely generated 
subgroup with 6r > 1/2. 

As Q is isotropic over Q, we have a Q-rational covering SL2 — > SOq. Therefore 
we may assume without loss of generality that T is a finitely generated subgroup of 
SL 2 (Z). 



2.1. Uniform Spectral Gaps. For the application to sieving, Theorem 1.8 de- 
scribed in the introduction is insufficient. One requires uniformity along arithmetic 
progressions; hence we recall the notion of a spectral gap. 

Let T(q) denote the "congruence" subgroup of T of level q, 

T(q) := {7 e T : 7 = I(q)}. 
The inclusion of vector spaces 

L 2 (r\H) C L 2 (T(q)\M) 
induces the same inclusion on the spectral resolution of the Laplace operator: 

S P ec(r\H) c Spec(r(g)\H). 
Definition 2.1. The new spectrum 

Spec„ e jr(g)\H) 

at level q is defined to be the set of eigenvalues below 1/4 which are in Spec(r(q)\H) 
but not in Spec(r\H). 
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Definition 2.2. A number 9 in the interval l/2<6*<<5is called a spectral gap for 
F if there exists a ramification number *B > 1 such that for any square-free 

q = q'q" with q' | «B and (q" , <8) = 1, 

we have 

Spec(r(g)\H) new n (0, 9(1 - 9)) c Spec(r(</)\H) netu . 

That is, the eigenvalues below 9(1 — 9) which are new for T(q) are coming from 
the "bad" part q' of q. As 03 is a fixed integer depending only on T, there are only 
finitely many possibilities for its divisors q' . 

Collecting the results in [BG07, BGSQ8] and their extension from prime to 
square-free of jGam02j we have: 

Theorem 2.3 r |Oam02l lBG07t IBGS08Q . 

(1) For any finitely generated T < SOq(Z) with critical exponent 6 > 1/2, there 
exists some spectral gap 

1/2 < 9 < 6. 

(2) If 8 > 5/6 then 6 = 5/6 holds. 

2.2. Counting with Weights uniformly in Level. Allowing some "smoothing", 
one can count uniformly in cosets of orbits of level q with explicit error terms. We 
fix a non-zero vector Xo £ Z 3 with Q(xo) = and set 

o = x r. 

We denote by Nq the stabilizer subgroup of Xo in G := SL^M). Then 

No = gaNgo 1 

for some go £ SL 2 (Q), where N denotes the upper triangular subgroup of G. 

Set K := g S0 2 (M)g ( 7 1 and choose n > so that a if -invariant rj neighborhood 
U v of e in G injects to T\G. Let tp := ip n be a non-negative smooth /-Co-invariant 
function on G supported in U v with J i/jdg = 1. 

Denote by Bt a KQ-'mvax\aiA norm ball in IR 3 about the origin with radius T. 

Definition 2.4. The weight £r : — > K> is defined as follows: 

6r(x) = / XT(xg)ijj(g)dg 
Jg 

where xt denotes the characteristic function of By . 

The sum of £t over O is precisely a smoothed count for #0 n Bt satisfying: 

^&(x)x#ons T . 

xGO 

Theorem 2.5. Let 6* 6e i/ie spectral gap for Y. 
(1) As T ^ oo, 

S(T) :^eTW~cT J , 

/or some c > 0. 
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(2) For square-free q, write q — q' q" with q' | 03 and (</', 05) = 1. Let Ti(q) be 
any group satisfying 

T(q)cT 1 (q)cT. 
Let No be the stabilizer of Xq in G, and assume that 

ri(g)n7V = rn% 

Fix any 70 £ T and e > 0. Then as T — » 00, 

£ Mx 7 o) = [r ^ M -(s(T) + £(T, (? ',7o)) 



where the implied constant does not depend on q or 70 . Here the error term 
satisfies 

f(T,g',[ 7 o])«T^ 
/or some /ixed Q > 0, does not depend on q" , and depends only on the class 

[ 7o ] miWAr. 

Remark 2.6. Assuming that T n iV is a lattice in A^, [Kon09 gives the above 
uniform count with the last error term 

replaced by a best possible error of 

Ts logT. 



2.3. Zariski Density of Orbits of Pythagorean Triples. 

For simplicity, we will use the notation V(R) to denote the set of all integers 
having at most R prime divisors. 

Let x £ Z 3 be a non-zero Pythagorean triple on the cone 

Q(x) = x 2 + y 2 - z 2 = 

and r < SOq(Z) a non-elementary finitely generated subgroup. Set 

O := x • T. 

Given a polynomial F which is integral on O, our goal is to find "small" values 
for R — R(0, F), for which F "often" has at most R prime factors. 

In fact, when studying such thin orbits, the correct notion of "often" is not 
"infinitely often" , but instead one should require Zariski density. That is, the set of 
x £ O for which -F(x) £ V(R) should not lie on a proper subvariety of the smallest 
variety containing O. We illustrate this condition with the following examples. 
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2.3.1. Example I: Area. 

Recall that given any integral Pythagorean triple x = (x, y, z) which is also 
primitive (that is, there is no common divisor of x, y and z), there exist coprime 
integers u and v of opposite parity (one even, one odd) such that, possibly after 
switching or negating x and y, we have the ancient parametrization 

x = u 2 — v 2 7 y = 2uv, z — u 2 + v 2 . 

In fact, this is just a restatement of the group homomorphism SL2(K) — > SO(2, 1) 
given by 

ac _ M 

i — y I ab — cd bc + ad ab + cd 



c d 



ac + b d " 2 + b2 + c2 + rf2 



where SL(2,R) acts on (u,v) and SO(2, 1) acts on (x, y, z). 

Consider the "area" \xy of the triple x (which may be negative). It is elementary 
that the area is always divisible by 6, so the function 

^Ia(x) := ~xy = -(u + v)(u- v)uv (2.7) 

is integer- valued on O. 

Remark 2.8. As above, we insist that the polynomial F is integral on O, but it 
need not necessarily have integer coefficients. 



As (2.7) has four irreducible components, it is easy to show that there are only 
finitely many triples x for which ^(x) G "P(2), that is, the product of at most two 
primes. Restricting to a subvariety such as 

u = v + 1, 

it follows conjecturally from the Hardy-Littlewood fc-tuple conjectures |HL22j that 

— xy = — (u + v)(u — v)uv = —(2v + 1) • 1 • (v + 1) • v 
12 6 6 

will be the product of three primes for infinitely many v. 

Since the set of triples generated in this way lies on a subvariety, it is not Zariski 
dense. On the other hand, it was recognized in .BG S08j that the recent work of 
Green and Tao |GT09j proves the infinitude and Zariski density of the set of all 
primitive Pythagorean triples x for which -Fa(x) £ V{4), that is, has at most four 
prime factors. 

Remark 2.9. The results of Green- Tao do not apply to thin orbits, and neither 
do the conjectures of Hardy-Littlewood. Indeed, we conjecture that that if O is 
thin and Y has no unipotent elements (which would furnish an affine injection 
into 0), then there are only finitely-many points x for which -FU(x) G V(3)\ On 
the other hand, allowing 4 primes should lead to a Zariski dense set of triples x. 
Below, we exhibit certain thin orbits for which there is a Zariski dense set of x with 
fU(x) G V(25). 

Remark 2.10. The critical number, 4, of prime factors above is related to the sieve 



dimension for this pair (0,F). We return to this issue shortly, cf. Remark 2.12 
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— xyz has at most four prime factors 
— xyz has exactly five prime factors 
• QQ X y z ^ as s * x or more P r i me factors 



Figure 3. The full orbit of all primitive Pythagorean triples. 




xyz has at most four prime factors 
xyz has exactly five prime factors 
xyz has six or more prime factors 



FIGURE 4. A thin orbit O of Pythagorean triples. 



2.3.2. Example II: Product of Coordinates. Consider now the product of coordi- 
nates xyz for triples x G O. It is elementary that xyz is divisible by 60, so the 
function 

^c(x) := ^xyz = —(u + v)(u - v)uv{u 2 + v 2 ) (2.11) 
60 30 

is integer- valued. 

Now we note that as (2.111 has five irreducible components (and the sieve dimen- 
sion is five) . Therefore there are only finitely many triples x for which Fq (x) £ 1^(3). 
Restricting to a subvariety such as 

u = v + 3, 

it follows conjecturally from Schinzel's Hypothesis H [SS58j that 
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— (u + v)(u - v)uv(u 2 + v 2 ) = — {2v + 3) • (3) ■ (v + 3) • v ■ (2v 2 + 6v + 9) 

OU oU 

will be the product of four primes for infinitely many v. Again, this set is not 
Zariski dense in the cone. See Figure[3j where it is clear that such points frolic near 
the x or y axes. 

On the other hand, it is a folklore conjecture that the set of triples x for which 
-Fc(x) G 7-* (5) spreads out in every direction. For the full orbit of all primitive 
Pythagorean triples (rather than a thin one), the best known bound for the num- 
ber of prime factors in Fq (x) follows from the Diamond-Halberstam-Richert sieve 
jDHR88llDH97] . Their work shows that F c (x) G V(YI) infinitely oftenj^] 

Again, when the orbit O is thin without affine injections, we conjecture that 
there will be only finitely many x for which Fc(x) G V{4) whereas five factors will 
be Zariski dense. Compare Figure[3]to Figure|4j We will exhibit certain thin orbits 
for which there is a Zariski dense set of x with Fc(x) G V(29). 

Remark 2.12. Sieve dimension is not merely a function of the polynomial F but 
really depends on the pair (0,F). In the related recent work [LS07 , Liu and 
Sarnak consider -Fc( x ) = xyz, where the orbit O = xo • T is generated from a point 
xo G Z 3 on a one- or two-sheeted hyperboloid Q(x) = t, where t ^ and Q is 
an indefinite integral ternary quadratic form which is anisotropic. Then the spin 
group of G = SOq(M) consists of the elements of norm one in a quaternion division 
algebra over Q, and T is the set of all such integral elements. 

In particular, their orbit is full, whereas the focus of this paper is on thin orbits. 
A common feature, though, is that there do not exist non-constant polynomial 
parametrizations of points in O (in our case this corresponds precisely to Nq l~l T 
being trivial). 

The sieve dimension for Liu-Sarnak's pair (0,Fq) is 3 (whereas in our case, the 
same function F = Fq has sieve dimension 5), and they prove the Zariski density 
of the set of points x G O for which -Fc( x ) is in "P(26). The precise definition of 
sieve dimension is given in Definition [2T61 

2.4. The Diamond-Halberstam-Richert Weighted Sieve. 

Let A = {a n } be a sequence of non- negative real numbers, all but finitely many 
of which are zero. 

For R > 1, let V(R) be the set of all integers having at most R prime divisors 
counted with multiplicity. Let W = {uj(n)} denote a certain sequence of weights 
supported on square-free numbers satisfying 

uj{n) < if n £ T(R), (2.13) 

and being of convolution type, that is 

uj{n) = ^w(d). (2.14) 

Let 

S(AW) :=^«„w(ii). 

n 



In fact they restrict to a subvariety in deriving the numer 17. 
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If we can construct a suitable sequence W with a good lower bound estimate for 
S(AW), then we can conclude by virtue of (2.13) that there are elements a„ G A 
with n G V(R), that is, having at most R prime divisors. 

Moreover, by ( |2.14 1 we can extract estimates for S(AW) from knowledge of the 
distribution of A along certain arithmetic progressions. For q > 1 a square-free 
integer, let 

A q := {a n G A : n = 0(g)} and \A q \ := a n , 

n=0(q) 

so that 

S(AW) = Y,"(<l)\A\- 
q 

Assume there exists an approximation X to \A\ := a n and a non-negative 
multiplicative function g(q) so that g(q)X is an approximation to |-AJ. Assume 
that g(l) = 1, g(g) G [0, 1) for q > 1, and that for constants K > 2, k > 1 we have 
the local density bound 

log z \ K f K 



Zi<p<z 

for any 2 < Z\ < z. 



Definition 2.16. The number k appearing in (2.15) is called the sieve dimension. 
(Note that it is not unique, as any larger value also satisfies (2.15); in practice one 
typically takes the least allowable value.) 

We require that the remainder terms 

r q := \A q \-g{q)X 

be small an average, that is, for some constants r G (0, 1) and A > 10 

E 4 ^ ) W<< 1 -^- (2-17) 

Finally, we introduce a parameter \x which controls the number of terms in A 
which are non-zero. Precisely, we require that 

max{n > 1 : a n ^ 0} < X Tfl . (2.18) 

We now state 

Theorem 2.19 ( [DHR881 IDH97] ). Let A, z, X, g, k, fjt and r be as described 
above. 

(1) Let o~ K (u) be the continuous solution of the differential- difference problem: 

(u-«a(u) = A-\ ybrO<«<2,^ = (2eT)T(/s+l), 

\(u- K a(u))' = -KU- K ~ x a{u- 2), foru>2, 

where 7 is the Euler constant. Then there exist two numbers a K and j3 K 
satisfying a K > (3 K > 2 such that the following simultaneous differential- 
difference system has continuous solutions F K (u) and f K (u) which satisfy 

F K (u) = 1 + 0(e" u ), /„(«) = 1 + 0(e~ u ), 

4 Recall that v(q) denotes the number of prime factors of q. 
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(2.21) 



and F K (resp. f K ) decreases (resp. increases) monotonically towards 1 as 
u — > oo; 

' F{u) = \/a K {u), forO<u<a K , 

/(«) = 0, for < u < p K1 

(u K F(u))' = ku k_1 /( u - 1), foru>a K , 

^(u K f(u))' = ku k ~ 1 F{u - 1), for u > a K . 
(2) For any two real numbers u and v with 

T^ 1 < U < V, f3 K < TV, 

there exist weights W = {cj(n)} such that 

S(AW)^>X J] (l-g(p)), 



provided that 

R > TflU — 1 



p<A-!/« 
v ju 



F k (tv-s) (l-^s) 



ds 
s 



(2.22) 



2.5. Statement of the Main Theorem. The following is the main result of this 
paper. 

Theorem 2.23. Let Q(x) = x 2 +y 2 — z 2 , xo G Z 3 a non-zero vector with Q(xq) = 0, 
and r < SOq(Z) a finitely generated subgroup with d > 1/2. Denote by 9 the 
spectral gap ofT. Let F be a polynomial which is integral on O, such that the pair 
(0,F) is strongly primitive. 
Then the following hold: 

(1) There are infinitely many x e O such that F(x) has at most R prime 
factors, where R{0,F) is given by (2.22) with 

r miii I „® , - - „ J I - "/"/ H > max ( - - n , - — ) . (2.24) 



(2) Under the further assumption that No D T is a lattice in Nq for Nq 



Stabc(xo) ; the bounds in (2.24) improve to 
5-6 



T < 



and 



fi > 



25 ' """" '"' 5 
(3) Denoting by k be the sieve dimension of(0,F), 

#{xGO:||x||<T,F(x)GP(i2)} 



(2.25) 



(logT)"' 

In particular, this set is Zariski dense in the cone Q = 0. 

2.6. Explicit Values of R(0, F). One must still work somewhat to obtain actual 
values of R from Theorem |2.23| We now state the smallest values of R(0, F) which 



are achieved from Theorem 2.23| at the expense of requiring the critical exponent 
5 to be close to 1. 



The first statement we give is unconditional. Assuming 5 > 5/6 and using 
Gamburd's spectral gap 6> = 5/6 from Theorem 2.3 the bounds (2.24) and (2.25) 



are equivalent. Hence assuming Nq fl T is a lattice in iVo and using the optimal 
error terms in |Kon09j does not improve the final values of R. Said another way, 
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the fact that our counting theorem is not optimal does not hurt the final values of 
R, unconditionally. 

In the next three theorems, we keep the notation Q,T,0 from Theorem |2.23| 
Let 



Fn(x) 



1 

GO 



xyz, 



and assume the pair (O, F) is strongly primitive. 
The following is the same as Theorem |1.5| 



Theorem 2.26. Let T have critical exponent 

5-h > 0.9992, 5 A > 0.99995, 5 C > 0.99677. 

Then the proportion o/x G O with F(x) € V(R) with ||x|| <T is 

1 



(logTp 



whe 



K A 



R H = 14, R A = 25, 



n c = 5, 
R c = 29. 



We now observe the effect of the worse error term on the quality of R(0,F), 
conditioned on an improved spectral gap. Assuming No fl T is a lattice in No, the 
counting theorem in Kon09j gives an optimal error term, which together with our 
sieve analysis gives the following. 

Theorem 2.27. Assume that No (IT is a lattice in No, and that the spectral gap 
9 can be arbitrarily close to 1/2. Then if 



5-h > 0.9265, S A > 0.98805, 
the conclusion of Theorem \2.2b] holds with 



5 C > 0.981675, 



Rn = G, 



R A = 14, R c = 17. 



Lastly, we demonstrate the values of R which we can obtain without assuming 
that r n Nq is a lattice in Nq. 

Theorem 2.28. We make no assumptions on Nq n T, but assume that 9 can be 
arbitrarily close to 1/2. Then if 

5-h > 0.991, 5 A > 0.97895, 5 C > 0.99905, 



the conclusion of Theorem 2.26 holds with 



R-h = 12, R A = 23, R c = 26. 



Theorems [2~26l |2~27] and [2~28| follow from Theorem [2~23] and Table [T] the com- 
putation of which is discussed in §5.3| 
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3. Equidistribution of Expanding Closed Horocycles 



Let G = SL(2,R). We keep the notation for N,A,K,n x ,a y , etc., from (1.10 1. 
We have the Cartan decomposition 

G = KA+K 

where A + := {a y : < y < 1}, as well as the Iwasawa decomposition G = NAK. 

Let r < G be a discrete finitely generated subgroup with critical exponent 
5 > 1/2. Assume the horocycle (N n T)\N is closed in T\G. In this section, we 
prove Theorem 1.11| 



3.1. Automorphic Representations and Spectral Bounds. Let 1/2 < s < 1, 

and consider the character \s on the upper-triangular subgroup B :— NA of G 
defined by 

Xs{na y ) = y s 

where a y = diag(y / y, ^/y^ 1 ) is as before, and n £ N. 

The unitarily induced representation (ir s :— Ind^y^jV,) admits a unique K- 
invariant unit vector, say v s . 

By the theory of spherical functions, 



fs(g) ■= (n s (g)v s ,v s ) = / v s (kg)dk 
Jk 



K 

is the unique bi if-invariant function of G with / s (e) = 1 and with Cf s = s(l — s)f s 
where C is the Casimir operator of G. Moreover, there exist some c s > and a > 
such that for all y small 

f s (a y ) = c s -y 1 - s (l + 0(y a )) 

by [GV881 4.6]. 

Since the Casimir operator is equal to the Laplace operator A on A"-invariant 
functions, this immediately implies the following. 

Theorem 3.1. Let 4> s € L 2 (T\G) K n C°°(L\G) satisfy A</> s = s(l - s)^ s and 
\\4>s\\2 — 1- T/ien there exist c s > and a > smc/i i/iai /or all y <^_\, 

(a y cf) s ,<i) s ) L 2 {T \ G) = c s ■ 2/ x - s (l + 0(y Q )). 

The irreducible unitary representations of G with if-fixed vector consist of 
principal series and the complimentary series. We use the parametrization of 
s € {1/2 + iM.} U [1/2, 1] so that the vertical line 1/2 + iW. corresponds to the 
principal series and the complimentary series is parametrized byl/2<s<l with 
s = 1 corresponding to the trivial representation. 

Let {Xi, X2, X$} denote an orthonormal basis of the (real) Lie algebra of G 
with respect to an Ad-invariant scalar product. For / 6 C°°(r\G) fl L 2 (T\G), we 
consider the following Sobolev norm S m (f): 

S m (f) = max{||A n • ■ •X im (/)|| 2 : 1 < ij < 3}. 

The following is well-known (cf. [CHH881 lKna86j ). 

Proposition 3.2. Let (71", V) be a representation ofG which does not weakly contain 
any complementary series representation V s . Then for any e > 0, and any smooth 
vectors vx,V2 £ V, 

\(n(a y )v 1 ,v 2 )\^ s y 1 ^- £ -\\S 1 (v 1 )\\-\\S 1 (v 2 )\\, as y -> 0. 
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3.2. Approximations to Integrals over closed horocycles. 

As S > 1/2, there exists a unique positive L 2 -eigenfunction <fi of the Laplace 
operator A = — y 2 (d xx + d yy ) on r\H with smallest eigenvalue 6(1 — S) and of unit 
norm |Pat76j . The spectrum of A acting on i 2 (r\H) has finitely many discrete 
points below 1/4 and is purely continuous above 1/4 [LP82J . 

Order the other discrete eigenvalues Xj = Sj(l — Sj), with Sj > 1/2, j = 1, . . . ,k 
and let <f>j denote the corresponding eigenfunction with \\4>j\\ = 1. For uniformity 
of notation, set sq := 5. 

The map g i— > g(i) gives an identification of G/K with EL Below a K- invariant 
function of G may be considered as a function on the upper half plane H by 
4>(x + iy) :— 4>(n x a y ) and vice versa. 

Let dg denote the Haar measure of G given by 

d(n x a y k) = y~ 2 dxdydk 

where dk is the probability Haar measure on K, and dx and dy are Lebesque 
measures. Hence for ipi,ip2 £ L 2 (T\G) K , 

(-01,-02)=/ 0i(s)02(.9) dg = ( t/j^x + iy)ip 2 (x + iy) dx-\, 
Jr\G Jr\u V 

Proposition 3.3. For any smooth ipx G L 2 (T\G) K and ^ 2 G C C °°(F\G), 
fc 

(ayTpi , 2 > = (0i » fa) (%<t>i » 02) + °e (y 1/2 ~ e Si (0i ) • Si (02)) ■ 

as y — > 0. 
Proof. Write 

i 2 (F\G) = VF Ao ®---®^ Afc ®y 
where W\ . is isomorphic as a G-representation to the complementary series repre- 
sentation V Sj , Xj = Sj(l — Sj), and V is tempered. Write 

0i = {ipi,<M<f>o H h (0i, 0fc)0fe + 0i*. 

Since the ^'s are the unique K- invariant vectors in W\ j (up to scalar), we have 



ijji € V K . Hence by Proposition 3.2 for any e > and y <C 1, 

K0i,02) = ^(0i,0 i )(aj / ^,0 2 ) + (a a Vi L ,02) 

3=0 
k 

= £>i > - 02) + O e (y 172 "^! (0! ) • 

3=0 

since 51(0^) <5i (0i ). □ 

The main goal of this subsection is to study the averages of the <f)j's along the 
translates (N n T)\Na y . 

We first need to recall some geometric facts. The limit set A(r) of F is the set 
of all accumulation points of an orbit T(zo) for some zq G H. As F is discrete, it 
easily follows that A(r) is a subset of doo(M.) = RU {oo}. 

Geometrically, N(i) C HI is a horocycle based at oo. Since F is finitely generated, 
the closedness of its projection to r\H implies either that oo is a parabolic fixed 
point, that is, NOT is non-trivial, or that oo lies outside the limit set A(r) [DalOOj. 
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We define for each < j < k 



4>^ \a y ) := j (j)j(nay)dn. (3.4) 

rnN\N 



Theorem 3.5. For any j = 0, . . . , fc, the integrals in (3.4) converge absolutely. 
Moreover there exist constants Cj and dj , depending on (j)j , such that 

<Pf{a y ) = Cjy 1 - 8 ' +d 3 y s i. 

Furthermore, Cq > 0. 



Proof. We first establish the convergence of the integral in ( 3.4 1 . If oo is a parabolic 
fixed point, that is, if N n T is a lattice in N, then the domain of this integral is 
finite, and of course the eigenfunctions of the Laplace operator are bounded, so we 
are done. 

On the other hand, if oo </ A(F), then Theorem A.l applies, that for y fixed, 

<f>j(n x a y ) < a (1 + x 2 )~ s i. 

Then the integral <j)j{n x a y )dx clearly converges, since Sj > 1/2. 
Since 

A4>j = Sj (l - Sj )4> 3 , 

it follows that 

The two independent solutions to this equation are y Sj and y 1 ~ Sj ■ 

Lastly, we must demonstrate that c > 0. If oo ^ A(T), this is done as in |Kon07j 
(see also [KO08, Equation (4.11)]), by proving explicitly that 

i.e. do = 0. As <po is a positive function, this implies cq > 0. 

If on the other hand oo is a parabolic fixed point for T, then as the Dirichlct 
domain for T is a finite sided polygon with oo as a vertex |Bea83j . it follows that 
for some Y > 1, (N n T)\N x [Y , oo) injects to r\H. Therefore, if we had c = 
and hence 4>o(n x a y ) — dgy Sr , then 

1 - ll^oll 2 > I" f \Un,a y )\ 2 dx% 

Jyq j v 

nj; £(Nnr)\N 

> [ ldn- [ d 2 y 2S - 2 dy = oo, 

J(Nnr)\N JYo 

since S > 1/2. 

This contradiction gives the desired result. □ 

The following Proposition shows that to compute (j)^ , it suffices to integrate over 
a bounded set J; the error term, if any, is small. 

If N nr is a lattice in N, set J := (NnT)\N. Otherwise as oo ^ A(r), A(r) is a 
compact subset of R, and we let J C (N D T)\N = K be an open bounded interval 
which contains A(T). In either case J is a bounded interval. 
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Proposition 3.6. We have 

0f (fly) = J H n * a v) dx + ( 3 - 7 ) 

as y — » 0. 

Proof. If N n r is a lattice in N, then <^ and cj>j are identical; there is nothing 
to prove. Otherwise, denoting by J c the complement of J, we have 

/ <j)j{n x a y )dn x < y Sj . 
Jn x eJ° 

by Theorem |A. 1| , which completes the proof. □ 

Next, we approximate (f)^ by smoothing further in a transverse direction. Denote 
by U e the ball of radius e about e in G. Let J be as in Proposition |3.6| 

Denoting by N~ the lower triangular subgroup of G, we note that NAN~ forms 
an open dense neighborhood of e in G. 

Definition 3.8. 

• We fix a non-negative function r\ € C^°(F n N\N) with 77 = 1 on J. 

• Let J7 Co denote the co-neighborhood of the identity e, and fix cq > so that 
the multiplication map 

supp(r?) x (U Co n AN~) -> supp(ri)(U C0 n A7V~) C F\G 

is a bijection onto its image. 

• For each e < cq, let r e be a non- negative smooth function in AN~ whose 
support is contained in 

W e := (u,nA)(u co nN-) 

and J w r e dv = 1. 

• We define the following function p v>e on T\G which is outside supp(r?)£/ Co , 
and for g — n x an~ € supp(r/)([/ Co n A7V~), 

p v ,e(g) ■= i]( n x) ® r e (an~). 
Proposition 3.9. We /icwe /or aZZ small < e <C eo arid /or aZZ < y < 1, 

l<£f K) - (a y 0j,P.),e)i 2 (r\G)l < e - y 1_Sj - 
Proof. This follows in the same way as Proposition 6.4 in K O08| . □ 

The next corollary follows immediately from Proposition |3.9| and Theorem |3.5| 
Corollary 3.10. For e < 1, and j = 0, 1, . . . , k, 

(ay(t>j,p n ,e)L2(r\G) < V 1 ^ 8 ' , 

as y — » 0. 
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3.3. Proof of Theorem The proof is almost identical to the proof of The- 

orem 6.1 in [KO 08 . We sketch the main steps. 

We use the following lemma, which is standard in Sobolev theory. 

Lemma 3.11. For ip G C^{T\G) K , there exists ift G C™(T\G) K such that 

(1) for any e < e and h G U t . 

\1>(9) ~ 4>W\ < e ■ for all g G T\G. 

(2) S m (ip$) <C S m+ i (tjj) for each m G N, where the implied constant depends 
only on supp(V>). 

Definition 3.12. For a given V G C°°(T\G) K and rj G C C (T n N\N), define the 
function T v (ip) on G by 



il)(na y )r)(n) dn. 



ne(rnN)\N 



Proposition 3.13. Let i/j G C°°(T\G) K . Then for any < y < 1 and any small 

€ > 0, 

\l v (ip)(a y ) - (a y ip,p v , e )\ < (e + y) ■ ^O")^), 
where tj)* is given by Lemma 



3.11 



Proof. This is the same as Proposition 6.6 in [KO08J. 
By Proposition |3.6| we have that 

0f(a y )=^(^)K) + o(y s O- 



□ 



For simplicity, we set p e = p^ jt where p„ e — rj <S> r e is defined as in Def. 3.8 Noting 
that r e is essentially an e-approximation only in the A-direction, we obtain that 
SM = 0(e- 3 / 2 ). 

Set p = 3/2. Fix i, a parameter to be chosen later. Setting ipo(g) ■— 4>(g), we 
define for 1 < i < £, inductively 



Ug) -=4-i(9) 



where ^>f_j is given by Lemma 



3.11 



Applying Proposition |3.13| to each ipi, we obtain for < i < I — 1 



lr,(ipi)(a v ) = (a y ^i,p € ) + 0((e + y)l v (t(; i+1 )(a y )) 



and 



2 n (ipe)(a y ) = (a y ip e , p e ) + 0{{e + y)Si(ipe,)). 
Note that by Corollary |3.3[ we have for each 1 < i < I 

k 

(a y ipi,p e } = 22(ipi,(t>j}(ay<f>j,Pe} + 0(y 1/2 Sx(p e )) 

3=0 
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Combining the above with Proposition |3.9| and Corollary |3.10[ we get that for 
any y < e, 

l n W(a y ) = (a y ^,p t )+o(^(a y ^ lPe }(e + y) k ^+0(S i+1 me + yY) 

k \ 

= ^(^0 J )K^,p e )+O(y 1 / 2 5 1 (p e )) + O hTy 1 ^ )+0(e e ) 

3=0 \i=l ) 

k 

= («v) + O (e • y 1 - 5 ) + O^y^e-P ) + 0{e% 

3=0 

where the implied constants depend on the Sobolev norms of ip. Balancing the first 
two error terms and recalling p = 3/2, one arrives at the optimal choice 

e = y («-l/2)/(l+p) < 

With this choice of e, the first two error terms are <C 6 y~ e . Lastly, we choose 
I to make the final error term of the same quality, namely £ = (4 — 36)/ (26 — 1). 
Therefore 

k 

il)(na y ) dn = V\^, <j>j)<j)f (a y ) + 0^, £ (y~ e ), 

l(NnT)\N J=0 

for any e > 0. 

This completes the proof of Theorem |1.11| 



/ 



4. Proofs of the Counting Theorems 

With Theorem 1 1 . 1 1 1 at hand, we now count establish the counting theorems; 
these are Theorems 11.81 and 12.51 



4.1. Proof of Theorem |1.8[ Recall that Q is a ternary indefinite quadratic form, 
SOq(R), and T a finitely generated discrete subgroup with 6r > 1/2. Fix a non-zero 
vector Xo £ K 3 , lying on the cone Q = such that the orbit O — xqT is discrete. 
Let 

jV(T) :=#{xeO: ||x|| < T} 

where || ■ || denotes a Euclidean norm on R 3 . 

Using the spin cover i : SL2 — > SOq, we may assume without loss of generality 
that T is a finitely generated subgroup of G :— SL2(M) with 6r > 1/2. We use 
the notation N, A, K, a y ,n x , etc from the introduction. Let dg denote the Haar 
measure given by 

d(n x a y k) = y~ 2 dxdydk 
where dk is the probability Haar measure on K, and dx and dy are Lebesque 
measures. 

As Q(x ) = 0, the stabilizer of x in G is conjugate to N and hence by replacing 
F with a conjugate if necessary, we may assume without loss of generality that N is 
precisely the stabilizer subgroup of x in G. It follows that x a y = y _1 x . Let Bt 
be a if-invariant ball of radius T in E 3 , and let \t be the characteristic function 
of this ball. Note that \t is right A"-invariant, that is, xrfe-gk) — xr(x-g), for any 
g € G and k 6 K. Also, as N is the stabilizer of x , we have xt( x o"-9) — Xt( x o<?)i 
for any n £ N, that is, xt is left A-invariant. 
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We define the following counting function on T\G: 

F r{g) ■= xt(x 73)- 

76Wnr\r 

Lemma 4.1. For any * e C C {T\G) K , 

(F T ,V)= / t>(n x a y )y- 2 dxdy. 

Jy>T-i||x j| Jn x ernN\N 

Proof. We observe that by unfolding and using the _ftT-invariance of \& and xb t , 

(F T ,9)= / XB T {*oig)^(g)dg 
Jr \G rnN\r 

XB T (*og)y(g)dg 

rruv\G 

/ 'fy (na y )y~ 2 dndy . 

x a a y \\<T JrnN\N 

As 'x.QOy = y Xq, the claim follows. □ 

As before, we order discrete eigenvalues < A < Ai < A 2 • • • < A^ < 1/4 of A 
on i 2 (r\H) with Aj = Sj(l - Sj), with s > 1/2, j = 1, . . . , k and let ^ € L 2 (r\H) 
denote the corresponding eigcnfunction with \\4>j\\ = 1. 

Recall by Theorem |3.5[ there exist constants Cj and rfj , depending on (f>j , such 
that 

(a y ) =c jV x - s i +d jV s K 

Furthermore, cq > 0. 

By inserting the asymptotic formula for JrniV\iV ^( na y)dn from Theorem jl. ll| 
we deduce: 

Proposition 4.2. for any * € C^{T\G) K and e > 0, 

(Ft, *>r\ G - E <t>j) r \ G (*. &>r\G + 0* (r*+f 

j=o v ' 

where the implied constant depends only on a Sobolev norm of "3/ and e. 

For all small rj > 0, consider an ry-neighborhood U v of e in G, which is K- 
invariant, such that for all T 1, 

B T U V C £(i+ n )T and B(i_ n )T C n u€Uv B T u. 

Let ^ £ C°°(EI) denote a non- negative function supported on {/,, with J G -i/Wg = 1. 
We lift tpr, to r\G by 

Then 

(F (1 _„ )T , *„) < F T (e) < (F il+V)T , * n ). (4.3) 
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On the other hand, recalling that {Xi, X2, X3} is an orthonormal basis for the 
Lie algebra g = sl(2, R) of G, we have 

($n,<f>j) = <t>j(e) + 0(ri sup sup X l cj) j (g)) 

gtUr, i=l,2,3 



where the implied consta nt is absolute. 
Therefore Proposition 



4.2 



yields for Cq 



0o(e)c o 



f >o, 



for some A > and any e > 0, where we used that the Sobolev norms of ^ v grow 
at most polynomially in 77; the implied constant now depends only on e. Therefore 
by equating the last two terms, we can choose r\ = T~ r for some r > and obtain 
from ((OJ) that 

F T (e) = C T 5 + 0(T S - C ) 
for some ( > 0. This proves Theorem |1.8| 



4.2. Proof of Theorem |2.5[ As in the discussion preceding Theorem [2~5| we may 
assume that P is a finitely generated subgroup of SL 2 (Z). We may also assume 
without loss of generality that go = e by replacing T by go^g^ 1 and hence the 
stabilizer subgroup of x in G is precisely the upper triangular subgroup N. Recall 
the weight £r defined in Definition |2.4| 

Let q > 1 be squarefree, and let Ti(q) be any group satisfying 

rWcr^cr 

and 

Nnr 1 (q) = Nnr, (4.4) 

where T(q) — {7 e P : 7 = /(g)} is the "congruence" subgroup of T of level g. 
We define the following if-invariant functions on r 1 (g)\G: 

F t(9) ■= E Xt(x 7.9), 
7 e(;vnr)\r 1 (g) 

and for any fixed 70 G L, 

n,(3):= E ^(7o _1 75)- 
7er l («) 

Proposition 4.5. PPe Ziave 

E ^(x7o) = <^,* 7o )r l(g) \ G - ( 4 - 6 ) 

xGx ri(q) 
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Proof. Starting with the left hand side of (4.6), we insert Definition 2.4 use that 
N stabilizes x , and that N ClT = N nT 1 (q): 

?t(x7o) = / XT{*&nog)ip{g)dg 

Y] / XT(^-oig)^{io 1 9)dg 



xex r 1 (q) xex ri(ij) 



7 6(wnr)\r 1 ( 9 )' 

FUg)^ho 1 g)dg 



E / nigmi^ig^g 

ieTl(q) Jv l{q) \G 



G 



F«(g)V JO (g)dg, 



where we used the definition of Fj,, "refolded" (the reverse of the "unfolding trick" ), 
and used the definition of \P!* . □ 



Applying Proposition 4.2 to each Ti(q) and noting that Sobolev norms of 



7o 



are same as that of ip, i.e., independent of 70 and q, we obtain: 
Proposition 4.7. Let 

0<6(1-S) = ># ) <\P <-<\% <l/4 

denote the point spectrum in L 2 (Ti (q)\G), and let (f>^ , . . . , (jyfi be the corresponding 
eigenfunctions of unit norm. Then for any e > 0, 



as T — > 00. The implied constant depends on e and a Sobolev norm of ip, but not 
on q or 70 . 

Lemma 4.8. For q'\q and (p( q € L 2 (ri(o/)\G) of norm one, consider the normal- 
ized old form <jM> m L 2 (Pi(g)\G) of level q' : 

6^ = 1 =6^K (4.9) 

TTien /or an?/ 70 € T, 

<^ ) .n,>r l(B )\G <* (,) >*?Wg 1 



10 /r 1 (q')\G \ /r 1 (q')\G 



(4.10) 
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Proof. Consider the inner product 

/ ri(g)\G JTi{q)\G 

4 q) (g)^ 1 g)dg 

" 7TO7OT/ ^'' (7MW(9)% 

where we unfolded, used (4.4) and the ri(</)-invariance of 4>^ q \ and refolded. 



Lemma 4.11. Let q = q'q" and assume that 
Then 

( ' F q (h {q} \ - 1 - ( A q,) T s i +/ q>) T 1_S M 

where c^ 9 ' and d q ^ are independent of q" . 

Proof. The inner product (^F^,(f)^^j can be unfolded again, giving 
F},4 q) ) = [ [ <j>f(na y )dn^ 

1 Jy>\\* \\/T J(Nnr)\N U 

1 r f ^\na v )dn% 



V^rW) : r i(9)] Jy>\\M/TJ(Nnr)\N 3 ^ "" V 
1 f (c^V^+d^V**)^ 

^[T-.T^q 1 )] f fM)..i-„ , >(9')„. S A^ 



□ 



vlrTT!^] 4>nx n |i/T V 3 3 Jy 2 

where we used Theorem |3.5| as well as the identity 

[r:r 1 ( g )] = [r:r 1 (g')][r 1 (g'):r 1 («)]. 

The claim follows from a simple computation and renaming the constants. □ 



Recall from Definition 2.2 that square- free q are to be decomposed as q = q'q" 
with q' | 03 and (q", 03) = 1. Let 

0<5(l-5) = A^ ) <A^<...<Ag ) <l/4 

be the eigenvalues of the Laplacian acting on L 2 (T(q)\M.). The eigenvalues below 
6(1 — 9) are all oldforms coming from level 1, with the possible exception of finitely 
many eigenvalues coming from level q' | 03. 

For ease of exposition, assume the spectrum below 6(1 — 6) consists of only the 
base eigenvalue Ao = 8(1 — 8) corresponding to 4>( q \ and one newform 4>^ from 
the "bad" level q' | 03. The general case is a finite sum of such terms. 
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Combining ( |4.6| ) and Proposition 4.7 with Lemmata 4.8 and 4.11 gives 

£ ^ - Ir7^M( F -*" , > rw (*" , -* , > rw 

+ [T:T 1 (q)] V T ' /t^gV ' ^/r l(9 ')\G 



Setting 



£(T,j,<y ) := [T : I\ («/)] (f t ,^\ , „ 



ri(«')\G \' 7o /r!(g')\G 



the proposition follows by recognizing the main term as the main contribution to 
H(T). 

This completes the proof of Theorem |2.5| 



5. Proofs of the Sieving Theorems 

We now consider 

Q = x 2 + y 2 - z 2 

and fix a finitely generated subgroup T < SOq(Z) with 5r > 1/2. Let xo G Z 3 \{0} 
with Q(x ) = 0. 

Again by considering the spin cover G := SL2 — > SOq over Q, we may assume 
without loss of generality that T is a finitely generated subgroup of SL2(Z). 
Let F be a polynomial which is integral on the orbit O = XoP. 

5.1. Strong Approximation. 

We first pass to a finite index subgroup of our original T which is chosen so that 
its projection to SL2(Z/pZ) is either the identity or all of SL2(Z/pZ). 

Lemma 5.1. There exists an integer N > 1 so that 

r(JV) = {7 € T : 7 = /(mod N)} 

projects onto SL2(Z/pZ) for p\ N . Obviously the projection ofT(N) in SL2(Z/pZ) 
for p\N is the identity. 

This follows from Strong Approximation; see, e.g. |Gam02l §2]. From now on 
we replace T by T(N). We use Goursat's Lemma (e.g. |Lan02| . p. 75) to have a 
similar statement for the reduction modulo a square-free parameter: 

Theorem 5.2 (Thm 2.1 of [BGS08] ). There exists a number <B such that if q = q'q" 
is square-free with (/|25 and {q" , 93) = 1 then the projection ofT in SL 2 (Z/qZ) is 
isomorphic to SL2(Z/q"Z). 



5.2. Executing the sieve. Choose a weight £t as in Definition |2.4| Let A 
A{T) = {a n (T)} where 

On(T):= J2 fr(x). 



F(X)=T. 



Then 

\A(T)\ = $> n (T) = ^ £ T (x) = S(T) =: *, 

n xeO 

where the leading term A" x T" 5 . 
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For q square-free 

\A q (T)\= Yl a «( T )= E 



n=0(q) 



Let r xo (g) be the subgroup of T which stabilizes x mod q, i.e. 

r x (<7) == {7 e T : x 7 = 7(g)}. 

Then 



\A(T)\ = 

Let 



-t'er XQ ( q )\r \ 7eStab r (x )\r 3C0 ( 9 ) 



\0 F (q)\ 




Ter xo (<r)\r 

-F(x 07 ) = 0(,) 



so that using Theorem 2.5 gives 

\A q (T)\ = \0 F (q)\ ( pJ {q)] (X + X ql ) + O e (T s \0 F (q)\(T e + TH!(M))) , 
with 

Xg> < X 1 - 6 , (5.3) 
for some C > 0, uniformly in q' . Define 

F r v l^(g)l 

Lemma 5.4. Assume the pair (0,F) is strongly primitive. Then for q square-free, 
g F (q) is completely multiplicative, and g F (p) < 1. Furthermore, 

(1) For F^(x) = z, we ftave 

I U, otherwise. 

(2) For F = F A = ±xy, we have g FA (p) = ip' 1 + 0(p~ 3/2 ). 

(3) For F = Fq = ^xyz, we have 

9 {P) -\6p-i+0(p-^), if p = 3(4). (5 - 6) 

Proof. For unramified q = qiq2 with qi and 92 relatively prime and both prime to 93, 
then 0(q), the orbit of x mod q is equal to 0(qx) x C(q 2 ) in (Z/qiZ) 3 x (Z/q 2 Z) 3 = 
(Z/gZ) 3 . 

For ramified q = g'g", with o/"|93, then T projects onto the identity mod q" , so 
0(q") is just one point, i.e. 0(q") — {x }. It also follows in this case that F (q'q") 
is isomorphic to O p {q') x F (q"). 

Since [r : r x „(g)] = 10(9)1, we have shown that 

g F (q) = l^li 
9 W \0(q)\ 

is multiplicative, and thus is determined by its values on the primes (only square- 
free q are ever used). 
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From the assumption that the pair (0, F) is strongly primitive, it immediately 
follows that \0 F (p)\ < \0(p)\, i.e. g F {p) < 1. Notice that if p|«8 then \0(p)\ = 1 
and \O f ( P )\ = 0, so g F (p)=0. 

It remains to compute the values of g F on primes p | 03. Denote by V the cone 
defined by Q — x 2 + y 2 ~ z 2 — 0, minus the origin. I.e. 

V = {(x, y, z) ^ (0, 0, 0) : x 2 + y 2 - z 2 = 0}. 



For F = F H = z, let 

ffi^xeF: F«(x) = 0} = {(a;, y, 0) ^ (0, 0, 0) : x 2 + y 2 = 0}. 

As V is a homogeneous space of G with a connected stabilizer, we have 

0(p) = V(¥ p ), and hence Fu (p) = Wy (F p ). 

We can easily calculate |F(F p )| = p 2 + 0(p 3 / 2 ). Up= 3(4), then Wi(¥ p ) is empty. 
If p = 1(4), then is the disjoint union of the two lines {(x, y) ^ : x = ±v— Ty}, 
each of cardinality p — 1. This proves claim (1). 
For _F = Fa = j^xy, we set 

W 2 := {x e V : F A (x) = 0} 

= {(0, y, z) ± (0, 0, 0) : y 2 - z 2 = 0} U {(a, 0, z) ^ (0, 0, 0) : x 2 - z 2 = 0} 

Thus Wi is the disjoint union of four lines, x = ±z, y = ±z, proving claim (2). 
For F = Fq = Q^xyz, we see immediately that 

W 3 :={xeV : F c (x) = 0} = Wi U W 2 , 

proving claim (3). □ 



From Lemma 5.4 the computation leading to (2.15) is a classical exercise (see 
e.g. |Lan53| ) . with sieve dimensions 



k = 1, 4 and 5 for F = i 7 ^, i 7 ^ and Fc, respectively. 



(5.7) 



Define r q := \0 {q)\T +£ . From the proof of Lemma 5.4 we have \0 (p)\ <C p, 



q squarcir. 



As X ~ cT , this error term is admissible, that is, satisfies (2.17), for any 



T < 



26 



The elements a n (T) are zero for n > T > X 1 / 5 , so pT8l) is satisfied for 

2 



(5. 



(5.9) 



We are not yet ready to apply Theorem |2.19| since our sequence A satisfies 

\A q \ = g(q)X + g(q)X q , + r q , 
the middle term of which is a nuisance. We define a new sequence A' via 
S(A'W) = £ (s(g)* + r q ) = £ a>(n), 
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and notice that 



\S(AW) - S(A'W)\ = 



^2u(q)g(q)X q , 



« A" 1 -" 



(5.10) 



by virtue of (5.3 1 



Now we may apply Theorem 2.19 to S(A'W). For any R satisfying (2.22), have 

S(A'W) > 1 



log K A- 



where k is determined in (5.7), according to the choice of F £ {F-u, -F4, Fq). To- 



X 



gether with (5.10), we now have that 

^»log«*' 

as desired. Having verified the sieve axioms, the upper bound of the same order 
of magnitude follows from a standard application of a combinatorial sieve, see e.g. 
|Kon09i Theorem 2.5] where the details are carried out. 

5.3. Explicit values of R. It remains to determine values of R for which the 
above discussion holds. 



The values of a K and j3 K in Theorem 2.19 can be tabulated, see for instance 



Appendix III on p. 345 of [DHR88 . The sets A appearing in this paper have sieve 
dimensions k = 1,4 and 5; for these values, we have 



a 1 =/3 1 = 2, a 4 = 11.5317.., f3 4 = 9.0722. 



14.7735.., /3 5 = 11.5347... (5.11) 



We will also need precise estimates on the functions which appear in (2.22). Al- 



though these are difficult to extract by hand, the following procedure is quite effec- 
tive in practice. Usually, u is chosen so that tu is near 1, and v so that tv exceeds 
a K . Precisely, for any ( S (0,/3 K ), set 

ru=l + C-C/fl,, ™ = ft,/C + & - 1. 

Then by Halberstam-Richert |HR74j . equations (10.1.10), (10.2.4) and (10.2.7), we 
obtain 



v/u 



( u \ ds 

F k (tv - s) 1 s — < (k 

\ V / s 



Thus Theorem [2~19l holds with 

fl>A*(i + C-C/flO-i + (K + C)ioi 



Ac 

c 



C)iog T - K + C^. 



•Cfl-=:m(C), 



(5.12) 



for any < C < Pk,- After inputting the values of /i, k, a K and the minimum of 
m(£) is easily determined by hand or with computer assistance. 

For the function F-h = z, we have K = 1, and a± = 2 = j3\. The best value of 
R is obtained for <J—>1, where we can take 6> = 5/6. Then (5.9) gives /1 > 12, 



and we have collected everything required to compute the minimum of to(£) de- 
fined in ( 5.12[ ). We find that the minimum value is attained at £ = 0.1203.. with 
to(C) = 13.931... Thus R = 14 is the limit of our method. For 8 > 1 - 1/1250 
and = 5/6, we find the minimum value m(0.1198..) = 13.992, which still allows 
R = 14. For comparison, consider instead a finite co-volume congruence subgroup; 
then 8 = 1 and can take 9 = 39/64 by [KS03] . This gives m(0.238..) = 6.48.., 
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F 


(TnN)\N 


5 


9 


r* 


m(C) 


R 




Anv 


1 


5/6 


12 


13.93.. 


14 


fin 


Any 


n GGG9 

u.yyyz 


£ /ft 
o/o 


i o 


1 Q GG 

lo.yy.. 




14 


fin 


Any 


i 


QG /ft/1 

oy/04 


o.lz 


ft A Q 
0.48.. 


7 




Finite 


i 


1 /o 
1/2 


4 


O.ZZ.. 


6 




Hi ni i-f$ 
L HULL, 




l/Z 






6 




Tn ii ni fp 


1 


1/2 


10 


11.8.. 


12 




Infinite 


0.991 


1/2 


10.2 


11.9.. 


12 


Fa 
± A 


Anv 


1 


5/6 


12 


24.9.. 


25 


FA 


Any 


fl GGGG^ 

u.yyyyo 


0/0 


Iz.U 


9/1 G 

Z4.y.. 




25 


771 

FA 


Any 


i 
i 


QG /ft/1 

oy/04 


o.lz 


lo.O.. 


16 


771 

FA 


Finite 


i 


1 /O 

1/2 


1 

4 


lo.o.. 


14 


Fa 
± A 


Finite 


98805 


1/2 


4.1 


13.9.. 


14 


F A 


Infinite 


1 


1/2 


10 


22.4.. 


23 


Fa 


Infinite 


0.97895 


1/2 


10.4 


22.9.. 


23 


F c 


Any 


1 


5/6 


12 


28.7.. 


29 


F c 


Any 


0.99677 


5/6 


12.2 


28.99.. 




29 


F c 


Any 


1 


39/64 


5.12 


18.7.. 


19 


F c 


Finite 


1 


1/2 


4 


16.7.. 


17 


F c 


Finite 


0.981675 


1/2 


4.2 


16.99.. 


17 


F c 


Infinite 


1 


1/2 


10 


25.9.. 


26 


F c 


Infinite 


0.99905 


1/2 


10.02 


25.9.. 


26 



Table 1. Values of R depending on S, 9, and whether N n T is 
assumed to be a lattice in N. 



allowing R = 7. If one could take 9 arbitrarily close to 1/2, the above calculation 
gives ro(0.292..) = 5.216.., or R = 6. 

Table [T] summarizes the discussion above and extends it to the other choices 
Fa = Y2^y and Fc — ^xyz, with various possibilities for 5, 9, and whether T C\ N 
is a lattice in N. For comparison, we also show the spectral gap 9 — 39/64 [KS03 
for congruence subgroups of SL(2, Z). These values of R are precisely those quoted 
in Theorems 2.26 2.27 and|2.28| in particular proving Theorem 1.5 



Appendix A. Proof of Theorem 11.131 



Recall the notation (1.10). Let r < G be a discrete, finitely generated subgroup 
with critical exponent 8 > 1/2. Let <j> £ L 2 (r\H) be an eigenfunction of the hyper- 
bolic Laplace-Beltrami operator A with eigenvalue A = s(l — s) < 1/4 and s > 1/2. 



The aim of this section is to reproduce the proof of Theorem 1.13 which was 
demonstrated in Kon07j- We give the statement again. 
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Figure 5. A flare domain. 



Theorem A.l ( [Kon.07] ). Assume that the volume o/T\H is infinite, and that the 
horocycle (N l~l F)\N is closed and infinite. There exist xq > and yo < 1 such 
that if \x\ > xq and y < yo, then 

« ■ 

as \x\ — > oo and t/4 0. 



The proof of this fact is reminiscent of the arguments given in Patterson [Pat75 
and Lax-Phillips [LP82 showing that a square-integrable eigenfunction of the Lapla- 
cian acting on an infinite volume surface must have eigenvalue A < 1/4, i.e. the 
spectrum above 1/4 is purely continuous. The key ingredient is that being L 2 forces 
an asymptotic formula for the rate of decay of the eigenfunction as it approaches 
the free boundary in the flare. 

A.l. Fourier Expansion in the Flare. Recall that a "flare" in the fundamental 
domain is a region bounded by two geodesies, containing a free boundary. Con- 
cretely, after conjugation in SL 2 (1R), we can assume that our group T contains the 

/ Q \ 

fixed hyperbolic cyclic subgroup generated by the element p- : z i— > 



l/Vk 

kz. So a flare domain is isometric to a domain of the form {z : 1 < \z\ < k; < 
argz < a}, where a < n/2. See Fig. [5j 

Such a conjugation sends the point at infinity to some point £ G [1,«], and a 
horocycle at infinity to a circle h tangent to £. See Fig. [6] 

Let <p e i 2 (F\H), ||</>|| 2 = 1, with A</> = s(l - s)<j). We proceed with the Fourier 
development of <fi using polar coordinates in this domain. As we are in the flare, 
<P(kz) = 4>(z), k > 1. Write z = re 10 in polar coordinates and separate variables: 

0(z) = = /(r) 5 (0), 

with f(nr) — /(r). Expand / in a (logarithmic) Fourier series: 

<j)(r,6) = ^ 5 „(6>)e 27rmlosr/logK . 



AFFINE LINEAR SIEVE 



31 




Then the solution to the differential equation induced on g n is (see e.g. |Gam02| . 
pages 180-181): 

g n (0) =c n V^6Pl l (cos8), 

where c„ € C are some coefficients and is the associated Legendre function of 
the first kind with 

A 4 = \-a (A.2) 
1 2irin 



2 log K 

We have proved 

Proposition A. 3. There are some coefficients c n £ C such that 

tf>(r, 0)=J2 c n e 2milosr/losK V^9Pf{cos9) (A A) 

n 

in the flare, with 1 < r < k and < 9 < a < tt/2. 
Next we need bounds on the coefficients c n . 

A.2. Bounds on the Fourier Coefficients. 



Proposition A. 5. The coefficients c n in (A.4) satisfy 
asn->oo. The implied constant depends on a and k. 

Proof. In Cartesian coordinates z = x + iy, the Haar measure is dz = ^p^. In 
polar coordinates this becomes 
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Input the expansion (A.4| into ||</>|| = 1, and consider only the contribution from 
the flare domain. This gives 

i > I I I0M)I 2 ^ 



1 JO 



> 



i ogK ]T| c „| 2 f a \p>}[ 

log^|c„| 2 P \P£{ 



dd 



sin f 



n\l2 dO 

cos 9 2 — , 
sin 9 



where we have decreased the range of integration by positivity. 
By Stirling's formula and the values of \i and v in (A.2|, 

Y(y + n + 1) 



I>+1) 



> n 



for b» 1. The implied constants depend only on k. 
Next we record the elementary bound 



^ e 2-Kn6/ log K 

Finally we have the formula (see |GR07j p. 1003 #3), 
PS (cos 0) = 4= + c°e[(^ + |)g + (M-|)f] A + Q (I 

r(i/ + §) v 7 ^^ V 



(A.6) 



(A.7) 



(A. 



(A.9) 



'("+§) 
valid whenever 

(PI) > 1, 

(P2) |H » 

(P3) |argi/| <ir, 

(P4) 0<e<6»<7r-e, and 

(P5) H»|- 

The big-Oh constant is absolute, depending on the implied constants above. 

The conditions (PI) and (P2) are immediately satisfied from the values of fj, and 
v in ( |A.2 ) . The argument of v approaches | for n large, so (P3) is easily satisfied. 
We will use this formula for 9 in a fixed range away from zero, 9 G [a/2, a]. Thus 
(P4) is satisfied, and (P5) is equivalent to (PI). 

Since 9 is bounded away from zero, so is the factor sin (9 in the denominator of 
(|A~9|. Putting together (|A.7|), (fOJ) and (|A.9|) gives 

(A.10) 



IP* 1 (cos 61) | > n - s e 27r " e / log ' t > ii- s e™/ logK . 



Returning to (A.6), consider the contribution from just the Ath coefficient and 

(19 



use ( ATOl) 



1 > log«^| c „| 2 f |P£ 
•W 2 



(cos 6»)|^ 



smf 



> | CJV | S 



IP 



1/2-s 



a/2 



-1/2+27TJ 



AT/lo gK (cOS^)| 2 ^ 



> | CA r| 2 A- 2s e 2,rA,Q/losK , 

as A — > oo. 

This completes the proof of Proposition |A.5| 



□ 
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A. 3. Radial Bounds for the Eigenfunction. Next we get bounds on the eigen- 
function (f> as the angle 8 decreases to zero. 

Proposition A. 11. Let (f> be as above with eigenvalue A = s(l — s), s > 1/2. Then 
as 8 -> 0. 

Proof. For 9 small, 



and 

l-cosflxfl 2 . 

We require some more estimates. First we use the following standard bound on 
the Gauss hypergeometric series: 

abx . 



valid for 



In particular, with 



F(a,b,c;x) =l + 0{ 
(a + £){b + £) 



\x\ max 



(c + £)(! + 



< 



1 2nm 



a = —v = — — 



2 log K ' 
b = 1 + v = a, 

c = 1 — fi = — + s, and 



1 — cos ( 



the above gives: 



whenever 



1 — cos 9 , 
F(-u, 1 + 1/,1-fi; ) < 1, 



n < 



We require |GR07j p. 999 formula 8.702: 



For z — cos 0, we have 



1 + z 



r(i — ju) vi -z 



,t/2 



1 + z 
1 - z 



M/2 



F(—v, v + 1; 1 — fi: 



Q-V = 5)8-1/2 



1 - 2, 



so together with (A. 12 1 we arrive at 

P? (cos 9) < fl-" = 6> s ~ 1/2 

for n < i . 



The analysis leading to ( |A.9| ) also gives 

|P£(cas0)| < n -« e 2*nfl/i<w«^-Va 

in the range n> i 



(A.12) 



(A.13) 



(A.14) 
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Thus we split the Fourier series as follows: 

cj)(r : 8) = ^c„e 27rmlogr / logK v / shi0P^(cos/ 

n 

= E + E 

n<X n>X 

— Si + S l 2; 

with X x A. 

j 1 

On 5i we use the bound ( A.13|): 



I Si I < \cn\Vti*0\P!?(coBO)\ 

n<X 



by the exponential decay of c n (clearly the series converges). 



On 52, we use the bound ( |A.14 ) 



|5 2 | < Y, K\Vsfo6\P>?(co S 6)\. 

n>X 



« E e 



n>X 

— 7rn(a— 20) / log k 
n>X 

< exp(-7rA"(a - 20) / log k) 
<C exp(— -7ra/ log k), 



i 



since X 

Combining the exponential decay of S2 with the polynomial decay of 5i, we 
arrive at 

^M)«0 S , 

as 6 -> 0. 

This completes the proof of Proposition |A.11| □ 

A. 4. Proof of Theorem |A.1[ Finally, we return to cartesian coordinates to con- 
vert the bound above into Theorem | A. 1| We require the following geometric anal- 
ysis. Recall Fig. |6j where £ is a point on the free boundary of the fundamental 
domain, h is a horocycle tangent to £, and 6 is the angle between the real line and 
the line £ from zero to infinity intersecting the horocycle, h at a point C. 

To return to cartesian coordinates, we redraw our picture after the conformal 
mapping 

^ z + t 
z — > , 

which sends the triple (0, 00, £) H> (1, — l,oo). See Fig. [7] 
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Figure 7. The image of Fig. [6] after a conformal transformation 
sending £ i— > oo, i— > 1 , and oo i— > — 1. The point C is a point of 
intersection of h and £, and B is the intersection of h with the 
y-axis. 



A 
B 




D 


\ / h 











Figure 8. A cleaner version of Fig. [7] The point A is the center 
of the circle £, and D denotes the origin. 



Lines and circles are mapped to lines and circles, and angles of incidence are 
preserved. The horocycle tangent to £ is now the horizontal line, h (tangent to 
£, which has been mapped to infinity). Similarly, the line i from zero to infinity 
passing through the horocycle is now a circle passing through the same points, 
having the same angle of incidence, 9, with the real line. 

We reconstruct this configuration yet again in Fig. [8| Let A be the center of the 
circle £, having radius R — R(0), let B be the intersection of the horocycle h with 
the y-axis, C the intersection of the horocycle with the circle, and let D denote the 
origin. 

It is easy to see through elementary geometry (since ^40 is tangent to the circle) 
that angle OAD = 9. Looking at triangle OAD, we see that 

R sin 9 = W= 1. (A.15) 

Let x — x(9) represent the length of BC, and y = y(9) be the distance from B 
to D. We aim to compute the precise dependence of x and y on 9. 
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We collect two identities for AB: 



AB = AD - BD = R cos 6 — y, and 



AB 



AC 2 -BC 2 = R 2 -x 2 . 



This implies 

x 2 = R 2 - (Rcosd-y) 2 = R 2 sm 2 6 + 2Rycos9-y 2 1 
which together with ( A.15[ ) gives 

x 2 + y 2 — 1 cos 9 1 
2y ~~ sin0 ~ 0' 



(A.16) 



as 9 

Thus (A.16), together with Proposition A. 11 concludes the proof of Theorem 

rxu 
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